Evidence-based Practices In Schools Research Articles

Implementation fidelity, student outcomes, and cost-effectiveness of train-the-trainer strategies for Masters-level therapists in urban schools: results from a cluster randomized trial

BackgroundLittle is known about the effectiveness and cost-effectiveness of train-the-trainer implementation strategies in supporting mental health evidence-based practices in schools, and about the optimal level of support needed for TT strategies.MethodsThe current study is part of a larger type 2 hybrid cluster randomized controlled trial. It compares two train-the-trainer strategies, Train-the-Trainer (TT) and Train-the-Trainer plus ongoing consultation for trainers (TT +) on the delivery of a group cognitive behavioral treatment protocol for anxiety disorders. Participants were 33 therapists, 29 supervisors, and 125 students who were at risk for anxiety disorders from 22 urban schools. Implementation outcomes were implementation fidelity and treatment dosage. Student outcomes were child- and parent-reported symptoms of anxiety, child-reported symptoms of depression, and teacher-reported academic engagement. We estimated the cost of implementing the intervention in each condition and examined the probability that a support strategy for supervisors (TT vs TT +) is a good value for varying values of willingness to pay.ResultsTherapists in the TT and TT + conditions obtained similarly high implementation fidelity and students in the conditions received similar treatment dosages. A mixed effects modeling approach for student outcomes revealed time effects for symptoms of anxiety and depression reported by students, and emotional disaffection reported by teachers. There were no condition or condition × times effects. For both conditions, the time effects indicated an improvement from pre-treatment to post-treatment in symptoms of anxiety and depression and academic emotional engagement. The average cost of therapist, supervisor, and consultant time required to implement the intervention in each condition was $1002 for TT and $1431 for TT + (p = 0.01). There was a greater than 80% chance that TT was a good value compared to TT + for all values of willingness to pay per one-point improvement in anxiety scores.ConclusionsA TT implementation approach consisting of a thorough initial training workshop for therapists and supervisors as well as ongoing supervision for therapists resulted in adequate levels of fidelity and student outcomes but at a lower cost, compared to the TT + condition that also included ongoing external expert consultation for supervisors.Trial registrationClinicalTrials.gov identifier: NCT02651402.

Helping Educational Leaders Mobilize Evidence (HELM): The iterative redesign of the Leadership and Organizational Change for Implementation (LOCI) intervention for use in schools.

Few "intervention agnostic" strategies have been developed that can be applied to the broad array of evidence-based practices (EBPs) in schools. This paper describes two studies that reflect the initial iterative redesign phases of an effective leadership-focused implementation strategy-Leadership and Organizational Change for Implementation (LOCI)-to ensure its acceptability, feasibility, contextual appropriateness, and usability when used in elementary schools. Our redesigned strategy-Helping Educational Leaders Mobilize Evidence (HELM)-is designed to improve principals' use of strategic implementation leadership to support the adoption and high-fidelity delivery of a universal EBP to improve student outcomes. In Study 1, focus groups were conducted (n = 6) with 54 district administrators, principals, and teachers. Stakeholders provided input on the appropriateness of original LOCI components to maximize relevance and utility in schools. Transcripts were coded using conventional content analysis. Key themes referencing low appropriateness were summarized to inform LOCI adaptations. We then held a National Expert Summit (Study 2) with 15 research and practice experts. Participants provided feedback via a nominal group process (NGP; n = 6 groups) and hackathon (n = 4 groups). The research team rated each NGP suggestion for how actionable, impactful/effective, and feasible it was. We also coded hackathon notes for novel ideas or alignment with LOCI components. Study 1 suggestions included modifications to LOCI content and delivery. Study 2's NGP results revealed most recommendations to be actionable, impactful/effective, and feasible. Hackathon results surfaced two novel ideas (distributed leadership teams and leaders' knowledge to support educators EBP use) and several areas of alignment with LOCI components. Use of these iterative methods informed the redesign of LOCI and the development of HELM. Because it was collaboratively constructed, HELM has the potential to be an effective implementation strategy to support the use of universal EBP in schools.

Research Literacy in Education and the Implementation of Evidence-Based Practices in Schools

Increased focus on and interest in Evidence-Based Practices (EBP) in schools justify a stronger focus on research literacy and implementation knowledge. This article presents two complementary perspectives on the transfer of educational research to practice. The “research literacy approach” focuses on the general competence and capacity of teachers to critically assess research before it is put into practice. The “school-wide implementation approach” has a broader perspective including organizational factors and the context of teaching. A review group which examined relevant research literature on implementation in schools identified six core components that advanced EBP. An example of a school-wide implementation in Norway is presented, which illustrates how a school-wide model was put into practice based on guidelines produced by experts in the field. In conclusion, both the “research literacy” and the “whole-school” approach can advantageously strengthen their emphasis on implementation knowledge in order to promote EBP in schools.

What interventions are cost-effective? A systematic review of cost-effectiveness analyses of school-based programs from 2000 to 2020.

The extent to which evidence-based practices (EBPs) are considered cost-effective influences educators' adoption decisions. However, what it means to be cost-effective and how to interpret cost-effectiveness ratios may be unclear. This systematic review of cost-effectiveness analyses of EBPs in schools illuminates the many sources of variability in published estimates. Studies were limited to peer-reviewed, school-based studies conducted in the United States between 2000 and 2020. Seven studies examining eight programs were identified and then coded for program descriptions, outcomes, research designs, and cost-effectiveness methodology. Secondary analyses illustrated how published estimates can be adjusted to reduce methodological variability and increase the utility of comparisons. The small number of studies highlights the need for research to evaluate the cost-effectiveness of more EBPs. Implications for research and practice are discussed. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Implementation readiness for evidence-based autism practices in school systems.

The increase in the number of autistic children being identified has led to increased demand on public schools to provide high-quality services. Effectively scaling up evidence-based practice (EBP) use for autistic students is challenging, given the complicated organization of special education. Teachers have significant challenges implementing autism EBP with fidelity. Factors such as implementation leadership and climate and attitudes toward EBP are linked to successful EBP use and may vary at different levels of the education system. Examining mechanisms of successful implementation is a critical step to support scale-up. In this observational study, conducted from September 2018 to March 2020, California school personnel (n = 2273) at multiple levels of the system completed surveys related to implementation climate, leadership, and attitudes toward EBP. Data were collected throughout California at the Special Education Local Plan Areas, County Office of Education, and district and school levels from educators and administrators working in public schools supporting autistic students. Multi-level modeling was conducted to characterize implementation readiness. Overall, implementation climate and leadership scores are low across levels with regional levels rated more positively than districts or schools. Attitudes toward EBP were moderate, with those working in schools having the poorest ratings and specialists/trainers and related service providers (e.g., speech-language pathologists) having the highest ratings. Outcomes provide a unique opportunity to compare implementation factors across organizational levels with a large, statewide sample. These data provide guidance for developing implementation interventions at multiple levels of the education system to increase readiness for effective scale-up of autism EBP in schools. Personnel and leaders at different organizational levels may need differentiated training targeting improved implementation climate and leadership. Personnel within districts and schools may experience a particular benefit from leadership support for EBP implementation.

Teacher attitudes toward evidence-based practices: Exploratory and confirmatory analyses of the school-adapted evidence-based practice attitude scale.

The Evidence-Based Practice Attitudes Scale (EBPAS) is widely used in implementation research, but it has not been adapted and validated for use among general education teachers, who are most likely to deliver evidence-based prevention programs in schools, the most common setting where youth access social, emotional, and behavioral health services. School-based stakeholders and a research team comprised of experts in the implementation of evidence-based practices in schools adapted the EBPAS for teachers (the S-EBPAS). The adapted instrument was administered to a representative sample (n = 441) of general education teachers (grades K-5) to assess the reliability and internal consistency via factor analyses. The S-EBPAS included two forms (i.e., EBP-agnostic and EBP-specific item referents), therefore, a multiple-group confirmatory factor analysis (CFA) was also performed to establish measurement invariance between the two forms. After adaptation and refinement, a 9-item, 3-factor structure was confirmed, with the final model supporting three first-order factors that load onto a second-order factor capturing attitudes toward adopting evidence-based practices. Multiple-group CFA analyses of measurement invariance indicated there were no significant differences between the two forms. Overall, this study provides a brief, flexible instrument capturing attitudes toward adopting EBPs that has high reliability and internal consistency, which support its use among general education teachers in school settings implementing evidence-based practices. The Evidence-Based Practice Attitudes Scale (EBPAS) is a popular instrument for measuring attitudes toward evidence-based practices (EBPs). This instrument provides valuable information during implementation initiatives, such as whether providers or front-line implementers have favorable attitudes toward a given practice. The EBPAS has been used in many different settings, such as in community-based mental health clinics, medical hospitals, and in child welfare. However, it's use in schools has been limited, and it has not yet been tested with general education teachers, who are key implementers of evidence-based practices in schools. In order to trust that the scores from an instrument are accurate, it needs to be evaluated when scaling it out to new populations and settings. One popular method to determine this is to use factor analysis, which was employed in this study. This study fills the identified gap by assessing the reliability (i.e., accuracy) and internal consistency of the EBPAS among a representative sample of general education teachers. Findings from this study indicate that the school-adapted EBPAS (S-EBPAS) is a brief, nine-item instrument that provides a reliable estimate of teachers' attitudes toward evidence-based practices. Our results also provide evidence that the S-EBPAS can be used to capture attitudes toward specific EBPs as well as attitudes toward EBP-agnostic. This study provides a flexible instrument that can be used by school-based implementation researchers, practitioners, and intermediaries at multiple phases of implementation projects, such as when exploring a new EBP to adopt.

The Role of Sensemaking in the Organizational Consultation Process in Schools

ABSTRACT Research on organizational consultation in schools demonstrates increased implementation of evidence-based practices by consultees and improved outcomes for students. Implementation science increasingly has been adopted as a framework for organizational consultants to use to promote uptake of evidence-based practices in schools. Although research that identifies processes and skills that increase implementation of system-level interventions remains important, a focus on how educators make sense of and enact the practices in their context is needed. In this article, we discuss how consultants can build upon implementation science approaches by exploring educators’ sensemaking. We draw from other disciplines to illustrate how educators make sense of new practices and their application within local socio-political contexts. Additionally, the importance of communities of learners and teaming in facilitating and evaluating sensemaking is discussed. Finally, we call for consultants to take active roles in investigating and facilitating educators’ sensemaking.

Conceptualising translational research in schools: A systematic literature review

Since Stenhouse (1975) called for teachers to have ownership of research, there have been initial forays into teacher led research and evidence-based practice in schools. However, the notion of translational research in schools remains contested, in significant part because it still has not been rigorously conceptualised. This is an issue of increasing concern, in our rapidly developing world. Now, more than ever, we need research-informed education, to address the complex challenges faced by school communities, and those with the most significant role in supporting them: teachers. To address the gap in understanding how best to promote translational research in schools, this systematic literature review asked, what do innovative and impactful translational, teacher research infrastructures look like? By translational research, we mean a process of agentic and agonistic democracy within which teachers critically develop and, or use research to support their classroom practice. The purpose and focus of this paper therefore, is to present the current extent and form of translational research practices in schools by undertaking a comprehensive, systematic review of the published literature on the issue. We found that the potential for translational research in education can be considerably enhanced when five key themes are taken into consideration, these being: Teacher-Researcher Collaboration; Teachers as Researchers; Research Cultures in Schools; Teacher Agency; Sharing, Accessing and Utilizing Research. The notion of technology - as a theme in its own right - was notable by its absence. From the findings we have been able to propose a foundational framework of translational research in schools. To date there have been no other systematic literature reviews on translational research in education, nor any frameworks proposed; and thus this paper addresses a significant gap in the field.

Implementation of school-based services for students with autism: Barriers and facilitators across urban and rural districts and phases of implementation.

The law requires that schools use evidence-based practices to educate students with autism spectrum disorder. However, these practices are often not used, or are not used correctly in school programs. Understanding barriers and facilitators of use of evidence-based practices in schools will help improve the implementation process. This study uses focus groups to characterize how school-based providers representing urban or rural school districts perceive barriers and facilitators for implementing new practices for students with autism spectrum disorder. Guiding questions include the following: (1) Are contextual factors perceived as barriers or facilitators and how do these vary by district location? and (2) What are the key factors impacting implementation across the Exploration, Preparation, Implementation, and Sustainment phases? Focus group participants (n = 33) were service providers to children with autism spectrum disorder from urban- and rural-located school districts. Several personnel-related themes (attitudes and buy-in, knowledge and skills, staffing, and burnout) were shared by participants representing both urban and rural districts. However, some personnel-related themes and organizational factors were unique to rural or urban districts. For example, themes related to system and organizational factors (leadership approval, support and expectations, district structure, competing priorities, time for effective professional development, litigation and due process, and materials and resources) differed between the district locations. This project serves as an initial step in identifying implementation strategies that may improve the use of evidence-based practices in schools.

Efficacy of Therapeutic Recreation-Based Drumtastic® and Kid Yoga Rocks in an Early Childhood Public School Setting

The intent of this study was to compare drumming and yoga to regular classroom physical activity in a school setting. Twenty-nine early childhood students with Autism Spectrum Disorder (ASD) engaged in 5-weeks of TR led Kid Yoga Rocks (KYR; n = 10), Drumtastic ® (DR; n = 11), or Classroom/recess control group (CR; n = 8). Statistical analysis revealed significant positive changes pre-post in cognition (p < .001) and mood/fun (p =.003) within DR and KYR groups. Significant positive affect change resulted in DR and KYR vs. CR (p < .001); though no difference between DR and KYR. No significant change in socialization with others pre-post for all three groups. However, near significant between groups change for personal relationships for KYR and DR vs. CR which showed DR significantly better (p = .023). Thus, TR led evidence-based practice in schools as a related service can benefit children with ASD including problem solving, memory, relationship building, and expression of positive emotion. Implications for TR practice in school settings and further research on the effects of yoga and drumming are warrented as a result of this study.

Using Systems-Level Consultation to Establish Data Systems to Monitor Coaching in Schools: A Framework for Practice

ABSTRACT Coaching has been widely implemented to support the delivery of evidence-based practices in schools. Although coaching is effective in rigorous research studies, scholars question the effective translation of coaching in authentic schools. Data systems that monitor the fidelity and impact of coaching offer one solution to address three implementation challenges described in the literature: (1) operationalizing coaching, (2) ensuring sufficient resource allocation, and (3) training in-service coaches. The purpose of this article is to outline a framework to guide systems-level consultants as they collaborate with schools to establish data systems for coaching, which include coaching data, teacher or classroom instructional data, and student data, which can be analyzed within the problem-solving process to monitor the fidelity and progress of coaching. Finally, a case example illustrates how these three levels of data can capture the multi-faceted process of coaching to address implementation challenges that impede the impact of coaching in schools.

Statewide Interagency Collaboration to Support Evidence-Based Practice Scale Up: The California Autism Professional Training and Information Network (CAPTAIN)

ABSTRACT The California Autism Professional Training and Information Network (CAPTAIN) is a statewide interagency collaboration with the goal of scaling up the use of evidence-based practices (EBPs) for individuals with autism spectrum disorder (ASD). CAPTAIN began as a clinical initiative then further developed under the influence of implementation science methodology. The Exploration, Preparation, Implementation, and Sustainment framework (EPIS) has impacted targeted strategy use for this statewide scale-up of EBPs by informing the development of key partnerships, implementation goals, and collaborative processes within CAPTAIN. Currently, CAPTAIN has over 407 members representing 140 school and community agencies who provide training and coaching in EBP and meet regularly with regional teams. Outcome data indicate 51.9% of the members provide training and coaching to more than three direct service providers/programs per year. Primary barriers to implementation of EBP were time for training (25.6%), lack of substitute teachers (16.5%), and staff lacking foundational skills (11.5%). Facilitators of implementation and sustainment of the CAPTAIN model include active participation in effective dissemination practices, creative funding and leveraging of local resources, development of the regional collaboratives with an active membership, member commitment to EBP for ASD, and use of implementation science to identify and overcome barriers. The purpose of this paper is to highlight CAPTAIN as a model for statewide scale-up of EBP in schools as well as other community agencies. Although these efforts have focussed on EBP for ASD, the concepts, partnerships, and procedures will likely be transferable to other focal issues and may be generalized across service sectors.

A Cost Analysis of the Innovation–Decision Process of an Evidence-Based Practice in Schools

The translation of evidence-based practices (EBPs) to improve students’ social, emotional, behavioral, and academic outcomes into authentic school settings has posed significant challenges for both researchers and practitioners. Among the many barriers to the adoption and use of EBPs are their associated costs. This study presents a framework for integrating the diffusion of innovation theory into an economic evaluation utilizing a societal perspective, which affords the capturing of costs of all phases from adoption through implementation of EBPs for all stakeholders. We leverage cost data from one evidence-based math intervention to serve as a model for incorporating diffusion of innovation theory into cost estimation. We exemplify how costs are distributed across personnel and organizational structures over time and across different stages of diffusion. Results suggest that the distribution of costs changes across stages of the innovation decision-making process, and lack of attention to the pre-adoption costs may partly explain challenges in EBP adoption in schools. Implications for the translation of social, emotional, behavioral, and academic interventions in schools and suggestions for future research are discussed.

Variables Contributing to Abandonment and Readoption of SWPBIS

There is a dearth of research examining school and district factors associated with the abandonment of evidence-based practice in schools. In this mixed methods study, we surveyed 23 school personnel working with 30 schools that abandoned and then readopted school-wide positive behavioral interventions and supports (SWPBIS). Our aims were to identify critical features abandoned and perceived reasons for abandonment and readoption of SWPBIS practices. SWPBIS features most often abandoned included data-based decision making and school-wide acknowledgment systems. Lack of staff buy-in was the most commonly reported reason for abandonment, and a new school administrator was the most commonly reported reason for readoption. Implications for research and how schools could safeguard against abandonment of evidence-based practices are discussed.

Adoption of PBIS within school districts

Recent research in implementation science has focused on identifying factors that predict adoption of evidence-based practices in schools. Less attention has focused on examining the adoption of practices within districts. This study included a sequential cohort of 552 districts in 25 U.S. states adopting an evidence-based framework, school-wide positive behavioral interventions and supports (PBIS). We examined schools within districts reporting on PBIS fidelity during their first five years of PBIS initiatives. Latent change score and multi-level growth modeling were used to examine and predict the percent of district adoption of PBIS over time. Results showed a significant increase in the rate of district adoption over the first 4 years of the initiative, with a decrease in growth between years 4 and 5. District size, proportion of students receiving free or reduced lunch, and districts located in cities and towns were found to be significant predictors of the percent of schools adopting PBIS. Implications for future implementation research and strategies for increasing the adoption of PBIS are discussed.

Confirmatory factor analysis of the Evidence-Based Practice Attitudes Scale with school-based behavioral health consultants

BackgroundThe Evidence-Based Practice Attitude Scale (EBPAS) is a widely used tool, but it has not been adapted and validated for use in schools, the most common setting where youth access behavioral health services. This study examined the factor structure, psychometric properties, and criterion-related validity of the school-adapted EBPAS in a sample of school-based behavioral health consultants.MethodA research team comprised of experts in implementation of evidence-based practices in schools along with the original developer adapted the EBPAS for the school setting. The adapted instrument was administered to a representative sample (n = 196) of school-based behavioral health consultants to assess the reliability and structural validity via a series of confirmatory factor analyses.ResultsThe original EBPAS factor structure was confirmed, with the final model supporting four first-order factors that load onto a second-order factor capturing general attitudes toward evidence-based practice. Correlations among the subscales indicated both unique and shared variance. Correlations between EBPAS scores and consultant variables demonstrated differential criterion-related validity, with the total score and the Requirements and Openness subscales demonstrating the strongest correlations.ConclusionsThe adapted EBPAS performed well when administered to behavioral health consultants operating in the educator sector, supporting the relevance of assessing attitudes in school settings. Potential directions for future research and applications of the EBPAS in schools and other service sectors are discussed.

Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship

BackgroundA substantial literature has established the role of the inner organizational setting on the implementation of evidence-based practices in community contexts, but very little of this research has been extended to the education sector, one of the most common settings for the delivery of mental and behavioral health services to children and adolescents. The current study examined the factor structure, psychometric properties, and interrelations of an adapted set of pragmatic organizational instruments measuring key aspects of the organizational implementation context in schools: (1) strategic implementation leadership, (2) strategic implementation climate, and (3) implementation citizenship behavior.MethodThe Implementation Leadership Scale (ILS), Implementation Climate Scale (ICS), and Implementation Citizenship Behavior Scale (ICBS) were adapted by a research team that included the original scale authors and experts in the implementation of evidence-based practices in schools. These instruments were then administered to a geographically representative sample (n = 196) of school-based mental/behavioral health consultants to assess the reliability and structural validity via a series of confirmatory factor analyses.ResultsOverall, the original factor structures for the ILS, ICS, and ICBS were confirmed in the current sample. The one exception was poor functioning of the Rewards subscale of the ICS, which was removed in the final ICS model. Correlations among the revised measures, evaluated as part of an overarching model of the organizational implementation context, indicated both unique and shared variance.ConclusionsThe current analyses suggest strong applicability of the revised instruments to implementation of evidence-based mental and behavioral practices in the education sector. The one poorly functioning subscale (Rewards on the ICS) was attributed to typical educational policies that do not allow for individual financial incentives to personnel. Potential directions for future expansion, revision, and application of the instruments in schools are discussed.

Pebbles, rocks, and boulders: The implementation of a school-based social engagement intervention for children with autism.

Few evidence-based practices, defined as the use of empirically supported research and clinical expertise for children with autism, have been successfully implemented and sustained in schools. This study examined the perspectives of school personnel ( n = 39) on implementing a social engagement intervention for children with autism. Semi-structured interviews, informed by the Domitrovich et al. (2008) framework, were conducted. Participants were asked about (1) school factors that affect the general implementation of evidence-based practices, (2) their specific experiences implementing the social engagement intervention, and (3) barriers to and facilitators of implementing the social engagement intervention. Data were analyzed using an integrated approach. General (e.g. implementation process, leadership, support, and staff) and intervention-specific (e.g. staff, barriers, and facilitators) implementation themes were identified. These findings suggest that a variety of factors should be considered when implementing evidence-based practices in schools and that implementing social engagement interventions for children with autism may require additional specific support for implementation.

Perceived Enablers and Barriers Related to Sustainability of School-Wide Positive Behavioral Interventions and Supports

The purpose of this study was to identify the most important perceived enablers and barriers regarding sustainability of school-wide positive behavioral interventions and supports. School personnel representing 860 schools implementing or preparing to implement school-wide positive behavioral interventions and supports completed an open-ended survey of factors regarding its sustainability. Qualitative analyses were used to assess perceptions of the most important factors related to sustainability. Thematic analysis produced 13 themes regarding enablers and/or barriers. The most commonly cited enablers were staff buy-in, school administrator support, and consistency. The most commonly cited barriers were staff buy-in, resources: time, and resources: money. Results are discussed in terms of enhancing durability of evidence-based practices in schools.

Practitioner Perspectives of Implementing Check & Connect

While there is a growing reserve of evidence-based practices (EBPs) available to practitioners, much can be learned about how to implement EBPs in real-world settings. Evidence of the effects of a widely disseminated student engagement intervention, Check & Connect (C&C), is emerging yet little is known about the implementation of C&C in community-based settings. The purpose of the authors in this study was to examine practitioner attitudes and perspectives related to the C&C intervention and implementation to gain an understanding of core implementation components that facilitated or impeded implementation. A researcher-developed survey instrument was used to assess practitioner attitudes related to the C&C model and implementation among 14 school-based practitioners working in a dropout prevention program. Findings indicate that practitioners were highly positive about the C&C intervention and in their attitudes about implementing EBPs. Benefits of C&C identified by practitioners included increased relationship building with students, tracking students on a consistent and timely basis, and addressing attendance issues as a main focus of treatment. The most common implementation challenges were time constraints, paperwork, and targeting absentee students. These findings contribute to the emerging literature on C&C and the implementation of EBPs in schools and community-based settings.